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The Variance of Rasch Ability Estimates from 
Partially-Known Item Parameters 



Abstract 

A closed-form approximation is given for the variance of examinee proficiency 
estimates in the Rasch model for dichotomous items, under the condition that only 
estimates, rather than true values, of item difficulty parameters sre available. The term that 
must be added to the usual response-sampling variance is inversely proportional to both the 
number of examinees in the item calibration sample and the length of the test. Illustrative 
numerical values suggest that the impact of uncertainty about Rasch item parameters on 
subsequent estimates of examinee proficiencies is less than has been observed in the 2- and 
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Key words: Cohen's approximation, Rasch model, variance components 



5 



Partially Known Item Parameters 

1 



Introduction 

It is common practice in applications of item response theory (IRT) to estimate item 
parameters from a calibration sample of examinees, then use these estimates as if they were 
known true parameters when estimating the proficiency parameters of these and subsequent 
examinees. Tsutakawa and his colleagues have pointed out that ignoring the uncertainty 
about item parameters leads to inferential errors, and offered a Taylor series approximation 
to incorporate this uncertainty into proficiency estimates and statements about their 
precision. Results for the two-parameter logistic IRT model appear in Tsutakawa and 
Soltys (1988) and for the three-parameter logistic model appear in Tsutakawa and Johnson 
(1990). (Also see Lewis, 1985, for an alternative approach to the problem, and Mislevy, 
Sheehan, & Wingersky, in press, for the application of these ideas to test equating.) 
Wright and Panchapekesan (1969) present without derivation a closed-form approximation 
for the variance of examinee proficiency estimates under the Rasch model for dichotomous 
items (Rasch, 1960/1980). Utilizing a series of computing approximations and simplifying 
assumptions, the present paper gives a somewhat simpler approximation. Neither 
numerical accuracy nor practical application is the point of this presentation— the 
aforementioned alternatives are preferable in this regard. Rather, by giving an 
approximation with fewer "moving parts," the objective is to aid intuition into the sources, 
the structures, and the consequences of uncertainty. 

The Rasch Model 

The Rasch model for dichotomous items gives the probability of a correct response 
to Item j from an examinee with proficiency 6 as 

Pj(9)= «K»bj> . 

l+exp(6-bj) m 
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where bj is the difficulty parameter of Item j. Letting xj denote the response to Item j, 1 for 
correct and 0 for incorrect, the assumption of local independence gives the conditional 
probability of the response vector x=(xi,. . .,xl) on an L-item test as 



where b=(bi,...,bL). 

Given item difficulty parameters and a realization of x, (2) can be interpreted as a 
likelihood function for 0. Unless all the responses are correct or all are incorrect, it attains 



its maximum at a unique point, the maximum likelihood estimate (MLE) 6. The negative 



reciprocal of its second deriva ;ve at this point approximates the sampling variance of 6, or 



Var(9l9,b). Given response vectors from a calibration sample of N examinees, the product 
of expressions like (2) serves similarly as a basis for the estimation of b and the sampling 



variance of the resulting estimate b. 

Cohen's Approximations 

Cohen (1979) gives closed form approximations for 8, Var(8l6,b), b, and 



Var(blb) for long tests when both 0 and b are normally distributed, in the samples of 
examinees and items respectively. Under these circumstances, item difficulty parameters 
can be approximated by rescaling the logits, or log-odds, of item percents-correct in the 
calibration sample, and examinees' proficiencies can be approximated by rescaling logits of 
their total scores by functions of item parameters. Adapting Wright's (1977) presentation, 
we begin with a calibration data set from which zero and perfect scoring examinees and 
items with all or no correct responses have been excised, then calculate the following 
quantities: 




(2) 
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n r 



tj = ln 



VPj\ 



j / 



t=Itj/L 
j=l 

T = i(t r t.) 2 /(L-1) 

*-*(£) 

L-l 

y. = X n ry' /N 
r=l 

Y = In r (y r -y.) 2 /(N-l) 

r=l 

ii-TY/8.35/ 



the proportion of the calibration sample that 
answered Item j correctly, for j=l,...,L, 

the number of examinees with a total score of r, for 
r=l,...,L-l, 

the log ratio of wrong to right answers on Item j, 
the mean of tj over L items, 



the variance of tj over L items, 

the log ratio of right to wrong answers for score r, 

the mean of y r over N examinees, 

the variance of y r over N examinees, and 

an expansion factor, in which 1.7 is used to match 
the logistic and normal distributions, and 2.89= 1.7 2 
and 8.35=1.7 4 . 



Then, from Cohen (1979), item parameters can be approximated as follows: 

b>W(t r t.) 



(3) 



and 
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Var(bjlbj)«jL 



W 2 )-C) 



Pjd-Pj)/ N ' 



where Cj=W 2 /[Pj( 1-Pj)]. The sampling covariances among the parameters of different 
items tend toward zero as L increases, as long as the items are not too dissimilar (Mistevy, 
1981, shows that the error correlations among equivalent items are -1/(L- 1)). We shall 
therefore approximate the L-by-L covariance matrix of sampling errors among item 
parameters as simply 

Var(blb)= diag&v..,% . 

N N (4) 

If item parameters are known, examinee proficiencies can be similarly approximated 
by rescaling logits of their total scores. Define three additional quantities: 

b. - 1 bj/L 

J_1 the mean of bj over L items, 

L 

U = £(b r b.) 2 /(L-1) 

j= 1 the variance of bj over L items, and 

X = (l + U/2.89)>« an expansion factor. 

Then, for a total score of r, 

S-b. + Xln( i I ? ) = b. + Xl„( 1 fej, 
where p r =r/L, and 

Var(0ie,b) - X 2 (-L-| = l[— Xi_ 

\r(L-r)j Hp r (l- Pr )/- (6) 



(5) 
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Estimating Ability with Item Parameter Estimates 

When b is known, (5) gives a proficiency estimate from x, and (6), evaluated with 
the true value of 9, gives its sampling variance. What is the sampling variance of 6 when 
(5) is cwh-wed with b rather than b? In this presentation, this amounts to using item 
parameter estimates rather than true parameter values to calculate b. and X. The setup 
corresponds to a two-stage experiment: 

Stage 1 : A sample of N examinees is drawn from a normal population. They 

respond to the L items in the test. Their responses are used to calculate b 
via (3). 

Stage 2: A new examinee is administered the test and the test score r is observed. A 
proficiency estimate is calculated using (5) with b. We can write this 
estimate as 0(r,b) to emphasize that it is a function of two statistics—which, 
it will be noted, are independent, given b and 8. 
The variance of b in this setting can be decomposed into two terms: 

Var(0l0) = E^[var(0l9,b)] + Var^[E(9l0,b)] . (7) 

XV 

The first term on the right of (7) is the expectation of (6) as evaluated with b, over 
the distribution of b given b. It depends on b only through X 2 , or l+U/2.89, where U is 
the variance of the true item parameters. The expected variance of the item parameter 
estimates is the variance of the item parameters themselves plus the average estimation error 
variance, C./N=(ICj/L)/N. Thus, 



E^[var(6l9,b) 



lprO-Pr)/J 



1 ( l-KU+C,/N)/2.89 
L\ pr(l-pr) 
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( x2 1 




\Pr(l-Pr)J 


l + LN(2.89p r (l-p r )) 



(8) 

This is the sum of the error variance for 6 with known item parameters and a term 
depending on their error variance that is inversely proportional to both L and N. 

The second term in (7) concerns the variation in 9 associated with the expected test 
score r for the true 6, as induced by variation in b. This term can be approximated by the 
delta method as follows: 



Var£ 



E(0l9,b), 



38 ~ 96 
* — Var(blb)~ 

3b' 3b 



L 

j=l 



L 

= £ 



jae | 2 ~ 

k) var( « 

/a 



[by (4)] 



b+xln, ^lj 



= 2 , 



2.89 LX 



Var(bjlbj) 



N ' 



Approximating all Cjs by their average, say C. , leads to the follow^;; 'simplification: 

C 



Var£|E(0l0,b). 



LN 



1 + 



8.35 X 2 



(9) 



Substituting (8) and (9) back into (7) yields the final approximation 

U 



Var(6ie)~-U X 2 LjC^ 
L\pr(l-Pr)| LN 



1 + 



1 



l-Pr 



I 2.89 Pr(l-p r ) 8.35 X 2 
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* Var(0l0,b) + 



LN\ + 2.89p r (l-p r ) 



8.35 X 2 



(10) 



Thus the sampling variance of 0 with item parameter estimates can be approximated as the 
sum of its variance with true item parameters and a correction term. The following 
observations can be made about the correction term: 

1 . It is always positive. 

2 . It is inversely proportional to test length (L). 

3 . It is inversely proportional to the size of the examinee calibration sample (N). 

4. It is directly proportional to C.. Since 



where Y is the variance of the logits of item percents-correct and Pj is the percent- 
correct for Item j in the calibration sample, the impact of the uncertainty about item 
parameters is greater when... 

a. the items are more dispersed, so that Y departs from zero, or 

b . items are farther from 50-percent correct, so that Pj(l-Pj) terms fall farther 
from the maximum value of .25. 

5 . Values of 0 for which expected proportions of correct response are near 50-percent 
are less affected by uncertainty about item parameters than more extreme values of 
0. This follows from the second and third terms inside the brackets in (10): As p r 
approaches .5, the second term approaches its minimum value of 4/2.89, and, 
because the logit of .5 is zero, the third term approaches zero. 



( l+Y/2.89 ) 
c J1-TY/8.35J 

Pj(l-Pj) ' 
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Illustrative Numerical Values 

To provide a feel for the relative contributions of sources of uncertainty this section 
evaluates the preceding formulae with some illustrative values. Standard normal 
distributions are assumed for 0 and b in the item calibration sample, implying the following 

values for true parameter values: 

t. = y . = b. = 0 
T - Y - .743 
X = W= 1.160 
U= 1 . 

Equation (10) is used to approximate 0 sampling variances, for 0 = 0 and 2, for test 
lengths of 10, 20, 40, 80, and 160 items, after item calibration with examinee samples of 
50, 100, 250, 500, 1000, 2500, and 10000. For C, we employ a representative value, 
namely that for an item one standard deviation from the mean. For such an item, P is about 
.7, in which case 

C. = W 2 /(.7 x .3) = 6.40 . 

The values of p r that correspond to 0 = 0 and 2 are .50 and .85. 

Table 1 gives results for 0 = 0, in terms of approximate sampling variances and 
proportional increases in sampling variance over those based on known item parameters. 
The proportional increases in standard errors are the square roots of the values in the 
rightmost column. Table 2 gives similar results for 0 = 2. 

[ Tables 1 and 2 about here ] 
The most striking feature of these results is how small the corrections are. Even 
with a calibration sample of only 50 examinees estimation error variances for subsequent 0 

estimates increase by only about 5-percent. This contrasts with the increases of up to 30- 
percent Tsutakawa and Soltys (1988) observed in posterior variances for 0 with a 

• 13 
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calibration sample of 100 under the 2-parameter logistic model. Tsutakawa and Johnson's 
(1990) results for the 3-parameter logistic were even more extreme, with increases in 
posterior variances more than doubling at higher levels of proficiency with a calibration 
sample of 400. 

Conclusion 

Cohen's (1979) closed-form approximations for the parameters in the Rasch model 
support a closed-form approximation for the variance of these examinee proficiency 
estimates when they are calculated with item parameter estimates rather than true item 
parameter values. The approximation is the sum of the sampling variance with known item 
parameters and a correction term. The correction term is inversely proportional to both test 
length and the size of the examinee item-calibration sample. Illustrative numerical values 
suggests the additional variance is quite small, yielding increases in standard errors of less 
than 5-percent even with calibration samples of only 50 examinees. 
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Table 1 

Approximate Error Variances at 0=0 



L 



1t T 

N 


10 


20 


40 


80 


160 


Var(0ie,b) 
Var(0l9,b) 


50 


0.5689 


0.2845 


0.1422 


0.0711 


0.0356 


1.0567 


100 


0.5537 


0.2768 


0.1384 


0.0692 


0.0346 


1.0283 


250 


0.5445 


0.2723 


0.1361 


0.0681 


0.0340 


1.0113 


500 


0.5415 


0.2707 


0.1354 


0.0677 


0.0338 


1.0057 


1000 


0.5399 


0.2700 


0.1350 


0.0675 


0.0337 


1.0028 


2500 


0.5390 


0.2695 


0.1348 


0.0674 


0.0337 


1.0011 


10000 


0.5386 


0.2693 


0.1346 


0.0673 


0.0337 


1.0003 


oo 


0.5384 


0.2692 


0.1346 


0.0673 


0.0337 


1.0000 
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Table 2 

Approximate Error Variances at 8=2 



L 



N 


10 


20 


40 


80 


160 


Var<eie,b) 
Var(6ie,b) 


50 


1.1066 


0.5533 


0.2767 


0.1383 


0.0692 


1.0483 


100 


1.0812 


0.5406 


0.2703 


0.1351 


0.0676 


1.0241 


250 


1.0659 


0.5329 


0.2665 


0.1332 


0.0666 


1.0097 


500 


1.0608 


0.5304 


0.2652 


0.1326 


0.0663 


1.0048 


1000 


1.0582 


0.5291 


0.2646 


0.1323 


0.0661 


1.0024 


2500 


1.0567 


0.5284 


0.2642 


0.1321 


0.0660 


1.0010 


10000 


1.0559 


0.5280 


0.2640 


0.1320 


0.0660 


1.0002 


oo 


1.0557 


0.5278 


0.2639 


0.1320 


0.0660 


1.0000 
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